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The role of a program evaluation unit in the Illinois 
s State Educational Agency (SEA) as' it attempts to provide meaningful 
data for a variety of audiences abou't projects that have extremely 
shorty life is investigated. The focus is on tvo programs — migrant . 
programs'* and neglected/delinquent programs — funded under Title I # 
Public Law 89-750. A common dilemma of these programs is determining - 
the success of children whose attendance is of short duration (less 
than two months in the migrant programs and an. average of seven 
months in th^ neglected/delinquent programs) . Government guidelines 
had^influenced the SEA and Local Educational Agency (LEA) staff to • ' 
, use standardized tests <pn a pre and post basis. The- evaluation unit 
fdund such testing inappropriate because of time factors, student* x 
attendance patterns, and student language abilities. As soon as 
communication linkage developed, there was mutual agreement by» the 
state evaluation^ personnel and -field people that standardized tests 
provided mean4ngless data. It was concluded that the intervention of 
state data collection in local projects requires that the chief 
audience of \.he data shoula be the LEA's themselves. It is only in 
this way' that relevant information can be amassed that will lead to 
improvement of projects. Data must be collected in a variety of ways 
and using .a' variety of tactics. (RC) 
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For four years—1972, 197>£ 1974, and" 1? 75— an internal Program Evaluation Unit 
■ ip the Illinois Office^of Education has been responsible £o^ % compiling annual 
-evaluation reports oJTthe use of fedetal funds fiqr compensatory education pro- ' 
grants in the state,' Through this experience the evaluation unit hes 'become 
acutely ^ware o£ the difficulties of providing meaningfyi evaluative, data to - 
meet .the legitimate request by Federal Government for annual evaluation reports.. 
This- paper dtiteflds to depict the role of. an internal evaluation unit in evalu^ 
ating two programs— migrant programs and neglected/delinquent, programs—funded 
under Title I, Public Law 89-750.' A comdon dilemma faced by trhe^e programs is 
determining the 'success of children whose attendance* is uf shor^ dutation (less* 
thai)/ two months in the migrarrt programs and an average of sevejl months in the 
neglected/delinquent programs). r 

The following 1 discussion includes a common introduction to the concept cf com- 
pensatory education; a 'description of toe responsibilities assumed by the Evalu- 
ation unit in the Illinois' Office o\ Education; separate 'discussions of. v migrant 
and neglected/delinquent programs illustrating the activities of the* evaluation 
unit; and a conclusion dealing with the role 6f an evaluation unit. at the State 
level. ' ' ' , * 



Compensatory Education ' • , ~ * - - . ( <, 

Both Title I 89-750 Migrant and Jitle I 89-750 Neglected/Delinquent are- considered 
to belong fto the generic category of compensatory education'. The philosophy undezfa 
lying compensatory education- deals with social justice; ■ * ✓ 

The central thrust of ESEA is td eliminate pdverty. The ^ ^ , ' 

; * underlying notion was. faniliar--pcor children given the * • * 4 >\ 

( opportunity t<> r W well in school will do well as adults...; * ' - 

titurphy, 1971) ' , . *• • . 

Compensatory Education assumes that schools can play' a: m^jor tole in improving J 
the academic performance of students in the lower end of the educational achdeve- 
ment spectrum. • Compensatory Education attempts to facilitate this through the, 
application of extra funds, specially designated personnel, equipment, and material 
to' the education of disadvantaged children. Along with the extra funds, is the • 
requirement "by the federal Government for annual reports from the states. Federal 
guidelines indicated that the evaluation should take place at least annually, that 
evaluation would indicate effectiveness of programs in meeting specific edu- 
cational needs of disadvantaged children, and that M ef f ectiveness" be determined 
through the. application of appropriate objective measures. 



Such a requirement^ often inappropriate in both the Title I 89-750 Migrant and 
Title I 89-750 Neglected/Delinquent programs in Illinois. Both of these programs 
involve children who enter and leave the programs in unpredictabltTlpatterns , attending 
for short amounts of time, and having reading and language j^atterns that -limit 
success on verbal measures of progress. 




The evaluation specifications stated aboveN^uld possibly be met in ye^r-long pro- 
grams through traditional tactics -such as pre- and po&t-tfesting with alternate forms 
of standardized instrumentation. However, in short-term projects such as these, 
the'' psychometric hazards 'inherent in testing are likely to be magnified, Sources 
of error in testing have long been known to psychometricians (Cronbacji, ' 1969) ; how- 
ever, the sophi^stication-of educators in schools and classrooms rarely approaches 
such sophistication (Hotvedt, 1974). Title I projects in attempting to comply with 
the federal requirement for "evaluation 1 ' based on "objective measures" are likely 
"to consider the "hard-data" of student gain to be an appropriate determinate 
of the success of programs. Indeed, requests from the State Offipe for Title I 
39-750 evaluative data from projects in Illinois has tended to reinforce such 
beliefs. ^ 

V 

StaR^(1972) and Vardrop (1971) have provided v|Jl considered critiques of student 
gain data? 'Tested**' hot measure learning, but father correlates of learning; and 
such correlates are not.Vdirect evidence of achievement. Stake (1972) points that 
the.correrarticnj of tests scores with performance, ori many specific educational tasks 
is seldom high. ' " **\ 

-f "~e -really wcii to <ncv whether or not a child i$ readina 
' at age-level^ we have a reading specialist listen to him " % 

^eai. She observes his, vending tuzbiist She might test him * * 
with recognition^ syntactic decoding^ and paragraph- ' t ° 

acrrprekeKsior. exercises. She would revest where 'evidence • * \ v 
^ was inconclusive. Sj^e would talk to nis teachers and his 
parents. She would- arrive at a clinical description which 
'. > <rLghz be reducablf to such q statement as "Yes, Johnny is 
* -reading ate*? abpve age-level* ft . > \ 1 - 

'The soqves we feet -from group reading tests can be considered 
4 estimates of /such a clinical judgment.. > These test "scores 
. 'correlate positively with the rrotfe valid' clinical judgments. 

Though m&rb objective, suph estimates ore 'not direct measure- * N 
penis of Dhai teachers or laymen mean by K '"the ability to' 
re . a A' " Aokiei&ner.1> gains for^a sizable nvnber^of students 
h ^ will be poorly estimated by them. . . . 

Stake and Vardrop (19?!) point out that error alone' could .show that a quarter of 1 
the students tested' by ^standardized achievement tests could show a gain of at least * 
bile year on a tfetest simply due to the-errors of '-measurement of- the test. The table ' 
below was provided by Stake (1972)' to illustrate that "growth"' on most standardized 
t&sts i$ a matter" of only a few raw-score ^points* 




Gain in Items Right Needed to Aav^ice 
' One Year Gain Equivalent on Three Typical 




i 



Type of Test 


< 

. Grade Eq 
5.0 


uiyalent 
6.0 


Items NeedecTTo 
Improve One Year 
G.E. x 


Comprehensive Test of Basic Skills, 
Level 3: Reading Comprehension 


20 


23 - 


- / - 3 ' 


Metropolitan Achievement Test., 
Intermediate Form 3: Spelling 


24 


31 


c 

7 


Iowa Tests of Basic Skills, 
Test A 1: Arithmetic Concepts 


10 


1* t 




Stanford Achievement Test, Form W,** 
Intermediate II: Word Meaning s 


18 


26 


• 8 



"More and more over the vpast few years,- criterion referenced measures have been 
utilized in the Title I 89-750 Migrant and Title I 89-750 Neglected/Delinquent 
projects as the "objective" measure determining success. However results of 
criterion referenced tests also are questionable as evaluations of student success: 
1) Criterion referenced tests may report immediate recall, but do not indicate 
student ability to relearn a skill once it has fad^ed from memory, 2) It is unreal- 

r istic to expect criterion referenced tests to deal with, each of the complex perform- 
ances expected as the result of learning/ 3) To be fair to the program, the testing 

• needs to b£ reasonably close to the teaching—the current state of development in 
criterion referenced test does not always allow this. A) In addition, the skills 
depicted by criterion referenced tests do not develop very rapidly and for short- 
term arrangements real growth will be difficult to discern. (Stake 1971,) Also, 
considering the current state of the art, many criterion' referenced .tests materials 
require further work on construct validity te> determine whether* the given test 
items are measuring progress accurately. „ 

• , ■ * 

These ^ere but a few of the considerations in mind as the Program Evaluation Unit ' 
undertook to provide annual report's to the Federal Government from the state. In 
addition ^to accepting the objective measure data from projects and acting as a 
broker of dubious ijnpressions other approaches we're considered to be necessary. 
'Student outcome data is only a part of what ^program evaluation might be. 

The next section describes the structure and function of the Program Evaluation 
Unit in Illinois, < The following two sections 'depict the activities and approaches 
implemented by the unit in providing evaluative reports on migrant and neglected/ 
delinquent programs., h 

The Program Evaluation .Unit • 1 . 

Prior to 1972 reports from the state to the Federal Government 'were constructed by - 
third, party contractors. The data gathering tactie, "utilized was an annual /'self 
assessment" questionnaire asking individual projects to report" such information as - * 



J 



it success, inservice, and ' dissemination efforts. After review ef. past 
repo;T^ submitted by the third party contractor -state officials found that 
although s imch reports would comply with the federal requests for data the^e w^s 
liptle, £&*%rff m9 information that would provide insight into the nature of com- 
pensatory eduction .programs or the leadership provided by the ^ state office. 

Consequently an i^ernal evaluation unit was organized to provide continuity in 
evaluation from yea* to year, and to utilize a broader range of data gathering 
tactics on^a wider range o^ issues. Reports were to be concerned not only with 
student outcomes but project organization and administration at & local ^and state 

^ level. Reports were to be .recommendatory directed local, state, and federal 

, \audience^. v 



In ,order 't^fjUJ^ such* a role, the unit required autonotav frbm prograffi staff, 
and independence to report findings accurately and -Validly' whe^er thexwere 
favorable or unfavorable to any of the parties concerned. TVie goal vas edi- 
bility in spite of being associated with the state office. It tfas decidedtt^t 
the unit would have to be located outside of the program area, that it wo\rld 
have to, fexamine office administration as rigorously as it examined local projects^ 
and that it would maintain a distinction between evaluation and project monitoring. 
Evaluation was to be focused on. issues and questions of .worth; project monitoring 
could check the details of congruence between projects and proposals. Where as 
monitoring might conclude that a project ,that ha'd* been conducted according to 
plan was appropriately implemented, evaluation 'could also point out that the value 
of the project was impaired in several respects according to identified standards. 

It was in this spirit that the evaluation ynit undertook Co implement a variety 
of tactics in order'to tap multiple sources of data for evaluative purposes. 
General data to satisfy federal requirements of -information about student success, 
f inservice, and dissemination of project techniques is gathered annually via a Loca 
Educational Agency 'self report. quesionriaire\ Additional data is gathered by the 
evaluation unit through observation of programs, interviews with state officials, 
project personnel, studejnt^and others to illuminate evaluative issues. The two 
sections below sketch these approaches as implemented by the evaluation unit in 
'examining migrant and neglected/delinquent projects. 
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Compensatory Education: Title I '89-750 Migrant 

, i 
First, let us look at migrant- educatlori. In 1966, Title I of the Elementary and 
Secondary -Act of '1965 was amended to specifically include children of migratory 
workers. this amendment was added bemuse reported statistics from Title I indi-* 
cated that migrant childreri^ere not receiving supplemental help under Title I 
89-10, Most of the Title I 89-10 programs were designed for the resident deprived 
child. Special programs were not set "up ".to fceet the nfteds"o£ -~tJiB migrant child. 
Parents of migflrant^ildren were urfa^re of what the schools *pr community could 
offer then or their children J Amending 89-750 was added" to Title I to fill this 
gap. The, S&e'mdment provided that the migrant prdgram be national in scope, that 
it -be a [flexible program, and that iA^iviffual states, try to coordinate and coopera 
wi4!h other states who were trying, to educate migrant children. 




V 



While there are various migrant streams across the United States, this paper 
focuses on throne that flows from Texas to Illinois. These migrant children 
K rarely finish any semester in Texas schools; they often leave early in the spring 
and arrive back late in the fall. To compensate for this loss of time, schools 
in the north are reimbursed for the cost of add^g the students at the end of 
or beginning of a regular term semester. These are called regular term projects. 
This paper, however, concentrates pn Summer term, programs ; it is here that "the 
planning, time; and money 'from the federal, state, and local agencies are. focused. 

Ideally, migrant . children (usually ages 6-13) would be at the summer session for 
the entire eight. Weeks. This ideal makes the following assumptions; that the 
families do not arrive late from other work locations; that they do not leave 
during the middle of the -summer school term to go to Wisconsin or Michigan to 
work on with. crops; that they do not leave early for Texas; that the oldest chil- 
dren do notdecide to work in die fields; that the parents decide they even want 
their children to attend school during the summer; and that the children them- 
* selves decide they want to attend regularly. 

' If the children do attend on a fairly regular- basis for the eight weeks a decision 
--J^f . S t0 be ^ de as t0 what Programs/activities need to be provided." How much do 
^Cheseschildren know? What are their problems? The Migrant Student Record Trans- 
fer System. (MSRTS) was suppose to answer a basic need of rap£diUXaS6^tQ^duca- . 
N^Onal anXoersonal data including health information. The MSRTS is su^oted to N 
prWfl$e transit of data from sch6ol to school via a state terminal* and a n>&ional 
terminal at Llt<le Rook, Arkansas. Test data and information concerning general ' 
education attainment and interest would be' sent to a school as soon as the infor- 
mation was requestecX^from^the state and national terminal operators. While theV 
,inal judgement is .no* yet in, the MSRTS has^ not been able to provide ready access ' 
to meaningful educational- data for teachers. Generally, only- the health data has 
arrived irra format that is^seful to local schools-. This is unfortunate since, 
\ T'X, the acadeaic information could go a long way in providing Local Education 
-\ Agencies with the opportunity to > individualize Instruction. 

Ei^ht weeks, at the maximum,- is not a long time .to attempt to assist a child to 
compensate for his education problems. Unless information is available from 
previous year-s.or previous teachers at that same summer school site, the migrant 
. cnil-d may well fiad .himself undergoing a series of time consuming tests. 

Each project director and the staff must evaluate their program, The national migrant 
guidelines request the submission of. a local evaluation to the. State Education Agency, 
.as well as the submission of a state evaluation report to the United States Office 
of. Education in Washington, D.C. However, there are no specific guidelines as to ■ 
what evaluation procedures are necessary or eyen desirable. 

The evaluation procedures that will be discussed below generally describe the' j * • ■ 
evaluation techniques used in Illinois from FY 72 to FY 75. 

» * * * * 

The first (and unfortunately often the only) evaluation procedure used by many 
program people consists of the .usual pre- post-standardized tests. ' As previously. 
. noted, Robert Stake" and James Wardrop (1971) raise' excellent points about the 
general utility of relying too heavily on these tests. Common sense seens to 
also illustrate the fallacy of judging a project as successful because a -child's 
tests scores rose from 3.3 on the pre-test to 3.5 6n tqe post-test during a- two. 
month program'. 



In the ^absence of firm guidelines by the Federal Government f and the State Educa- 
tion Agency, the annual report questionnaire completed by each of the projects 
'shaped their behavior/ When th^ LEA report questionnaire that the migrant proj- ° 
ect sites used to record data from their FY 72 evaluation report to the State 
Education Agency itself is scrutinized, one notices that the emphasis is heavily 
orf standardized tests. The FY 72 pre-test and post-test dates'were asked for by 
grade level, the name of^ the test and the test battery are also requested. * 
Three of the summer pojects' used the wide range achievement test (WRAT) Two 
of these three projects expected students to achieve ^one we^k gain" on the WHAT 
for each week gf instruction. The' third projectSnrote, "It was most difficult 
to have a standard of success to assess results ^ue tp the attendance. of some 
children. Many students who took pre-tests were not In .school near tfre end of 
the program as they had started to work in the -fields:" ' Such-Tnsftruments used" 
the Metropolitan Achievement Jest, the Iowa Test of Basic Skills, and' the Veabody 
Picture Vocabulary Test. . k r' ' 9 

In FY 72 the annual report questionnaire had been desigr^d by a third party evalu- 
ator under contract to the State Education, Agency' while the report itself had been" 
written by the newly formed internal evaluation unit. The report had to emphasize 
,the great losses of data because there were so few sites that could "supply "pre- 
test and post-test standardized test data on the same children. . ** . 

In FY 73 and FY 74 the -internal evaluation unit developed the end of the year' * 
LEA report questionnaire whictf then listed standardized, tests as just one mSthoji * 
of evaluation along > **i£h teacher-made- tests , * observation, 1 ' and criterion refer- 
ence tests. -n. - ^* * 

The data reported back from* the sites, when analyzed either by project or- by 
separate subject area,, indicates that observation followed closely by teacher -made 
tests.were by 1 far the most frequently used methods of evaluation , during FY 73 and/ 
FY 74; standardized tests came in third. Interviews conducted by th# evaluation V 
unit with staff at the state and local levels over these years hav^ indicated - 
a general attitude that the standardized achievement test was a waste of tinfe 
and money for migrants 'children, 'Disregarding the question of instrument bias 
and whether migrant -children had the necessary English reading skills to make . 
the tests valid x *the v data just could not- be fed back quipkly enough from the 
pre-test; and nobody seemed to think that the post-tesfrs proved anything anyway." £ 
x . . „ * 

tua addition to data on student success gathered and reported by each of the Iqc^I 
projects, the evaluation unit decided'to use a-case study approach to attempt to 
describe what pro jects -were in. reality. One case study was' conducted by ,the,evalua 
tion unit to explore the contention of. state office personnel that "good migrant 
education projects had pbsitiye* affects on general community attitude .toward the 
migrant population. .In drder to test this hypothesis evaluation. uni.t 'staff visited 
what was considered to bq* ,an exemplar program. The evaluators observed >cj.asses , >, 
talked v^th s^udentrs, teacher-ii, the. "project director, the local" principal $nd. the 
superintendent, In^ addition,, the evalutors interviewed community members— the * 4 
pharmacist; a banker # a grocery sfor,e owner, clerks, gas station attendants, *and* % 
the manager of the canning factory complex thai t hired the migrants locally., This*" 
la-st 'individual had also*beeft on the school-board for 20 years T 4 ' 



the case study effort showed that the local summer school program f,or migrant 
children ha,d no effect on local attitude. The' Children had been segregated into 
a different 'part of the buildings from other students / there was no community 
concern about this policy. The community Vas content to maintain the police of f 
segregation. The quality of the local migrant ejSucatioil program was indepen^jit 
of community attitudes toward migrants in general* 

* 

Q » . 

Compensatory Education: * 89-750 N/D # * s 

In November >J.96'6, # Title I of the, Elementary an&'Secondary^Education Act of 1965 
was % amended *o include neglected or delinquent children living in state or private 
institutions 1 . ^ As in other Tit; 16 I programs, the* intent of the amendment was to 
provide supplemental educational services to a targeted portion of the children at 
^institutions. Targeted children ^were to, be the most educationally disadvantaged 
'children at the facility. Programs wqre to based ori specific assessments of the" 
'educational heeds of the deprived children and Were to be written in performance 
terms* in re fat; ion. to those, needs. ^(Prograri 'Guide 44 Regulation 116.18(b)) 

Ta£ge>t£d Title I students in the cfo^rectionai facilities in Illinois have been, 
described, in the following manner or. applications- for gfants: 



These students- ore lacking the fundamentals of ^reading: 7\ 1 
The students are lacking in, motivation and have* histories ^bf^ 
* , vds.t Siohoql failures and, truancy. * x * 

> * ■ * . \ • ' * 

v Az % the present time, -/.targeted Title I students are, func- \ 
• tioning four or more yzars below expected grade level... . * \ 
^These- students 'are between the ages of 15 and-l? e and are J 
' - functioning beldw ike sixth arade level and math levels. 

> ' w \ • - \ . ■ 

♦ ...Approximately 20% of the *■ students. are non r reader,s. \ . . 
^ This inability to read makes it impossible for them to. follow' 
a normal academic schedule. . . . " : 

At the present time 1 IS delinquent and/or socially maladjusted 
\ - % ypjwg m en m ages\lZ to ISP ape functioning five years or more 
below. thei*r expected reading -arts level*... 

At present^ 151 'delinquent poys ages 13 to 20. :-. score be.lcw 
4.P in math'' as keasitred^by^SAT.* They are six or more years 
below grade placement. 1 They have IQ f s below 90. 1 
\ ' \ 

' , . I. It is realistic to assum$- that^most, or all, of these youths 
\ ytZZ riqt pursue their academic "trqinihg 1 after parole. t ± \ 

j ' . . .About 7%. , , (these^ teer4g§ boys) are functioning extremely 

low (first grid sepond -grade levels f in reading and math 

'xany of ohese same beys are lacking in almost all soc&l . 
skills necessary to interact with their peers. ^Thie condition 
results iv^OK inability of these focy^' to function in a regit- * 
lor departmentalized Junior t High Scho'ol program. ' ' ° 1 , 



\ 



These excerpts from an actual application spell out a consistent picture of the 
adolescent deliquent— academic achievement is seriously limited. There are 
obviously toany/ other problems* to take into account, in the education of delinquent 
youth. Severe ,socio-environmental difficulties such as low evaluation of edtica- . 
tion, destructive and aggressive behavior in opposition to what, is often preferred 
as an educational opportunity.' 4 ' ' „ 

Further, recidivism is a fact,, of life for institutionalized student. fn 1972, - • 
rates ranged from a low of i% to a high of 83%. One institution lis/ted reasons 
for recidivism to include "boor parental supervision, parental rejection, -other 
family problems, peer delinquency, gang orientation^' community attitude, high ' • 
delinquency, criminal milu, and social economic conditions." 

Title I, ot any educational program, for such youth "requires much that is- not 
present. Academic endeavors require compliance, it ,is" jtist. such a lack that 
institutions most often point' to as Treason for' lack of student success. -Insti- 
tutions most frequently report "lfck of motivation,",' nonattendance (students are 
often- above Che legal schopl leaving age* and cannot be required to attend class) 
and leaving the program' prior to completion (remember the seven -month attendance / 
average) as the greatest "barriers to supcess.' This represents but a brief sketch 
of the setting for ppi^lems to.be faced by the federal requirement for evaluating 
Title I 89-750. ( * . * 5 ■ 

Federal requirements include t;he following: . ' 

That the local educational agency will make art* annual report * 
and such other reports to 'State educational agency > in such ' 
form and containing such information (which 'in the cas'e of 
reports relating ' 'to performance is in accordance with specific* ~ 
performance -criteria related in program objectives) > as may be • . 
Reasonably necessdry' to^enable \ tfce State educational agency, to. ° ' : 
**- > perform its duties under 'the title, including information 

relating* to the educational^ achievements of students partici- , 
* . pattng in programs carried* out under this titled and will 'k4?p 
> t - suck" records and afford such' access there to as the -state edu- 
. national agency may find necessary^ to assume the"' correctness'' x ' 
. t and verification of such reports,. /. . * ' 4 V 

That effective procedures $noltf8ing provision for appropriate 
^objective, measures 'of edud&tional achievement, will be adopted: 
for evaluating at least annually the effectiveness of the pro- * ' * 

grams vn. meeting specific educatioztal needs of educationally 
' ' deprived children. ' „ 

' v , ■ 

—Sectiqn Ul (g.1 (SV . " - ' . , - - , * ' \ 

In other words, fcased o,rf specifically stated student orientated pbjectives and , 
activities measured "objectively an annual report will indicate the effectiveness 
of programs in meeting specific educational needs. From 1972 through 1974. Illinois 
Reports to the Federal Government lamented the failure of objective writers to master 
the' art of setting standards: / 



An effort should be<made by the SEA to Ideate available , * 
tests- desptfhed spedifically for ,shorU term institutional- 
ized children'. ft972 page 100) 

...An effort should be made by the SEA to locate available 
test designed specifically for short-teim institutional- 
s ized children* (-1973'page 45) . ' * " 

»jf * • 

Yet testing pre- 1 and post-treatment td determine program success through ^student 
outcome measures is fraught with hazards and short-term programs . One teacher com- 
mented on 'the standard of .one month gain per one month in program: "I do not^ think 
this is realistic because it does dot take into consideration the student's back- 
ground, personality, needs, school history and social adjustment." (Hanna City 
page 14) Short-term test, re tea t % of such Title I students allows the possibility 
that regression effects an error ^score would give the appearance of gain where there 
may .not have been any, (Stake, Wardrop,\1971) Short-term evaluation using such 
tests as the Stanford Achievement Test or Wid^e Range Aptitude Test (as these programs 
typically 5 do) is not likely to accurately 'assess gain. 

Individualization is the > ^truction&l tactic most often reported by programs. If^ 
individualization means persb^^Lzed objectives, activities, and criteria for success, 
what manner of . determining. student success is appropriate? Gain scores on norm refer- 
enced achievement test do not measure success appropriately in these cases. The 
alternative suggested by the evaluation* unit was criterion referenced testing. (1973) 
Hbnever, irregular attendance, sudden leaving, and noncooperation are not matters that 
an? testing process can overcome. * t 

The next possibility suggested by the evaluation unit (1974) was that there should 
be follow-up studies oh samples of students to determine program effectiveness. « The 
•type of longitudinal study ^purposed by Wholey et al. (1973) served as the model 
for .these evaluative suggestions ; * 

v - ..\To look for longitudinal rather than short-term effectiveness. 

.To establish and revalidate the usefulness of short-term out * * 
put measures as* predictors of the values of long term out put 
measures. + 

...Tq sample program recipients seeking insights on the per- % 
. sistance or lack of persistance of program benefits. 

Yet in the- fevaluati6ri iinit t s < 1972 report, comments on recidivism indicated that pro- 
grams begun* for students in the institution had* no follow-up in the home community. 
Community schools were not anxious to accept the student upon release. A re-entry 
coordinator in the Chicago area has been seeking to alleviate such problems. A 
re-entry report dated April 25, 1976, indicated that only about one-third of paroled 
students re-enter public schools. This corresponds almost precisely , with legal ' * 
school leaving age. t The report indicates that, "School re-entry problems are related 
to several factors, the most prominent of which is the lack of planning for school 

rfe-entry prior to parole Another factor affecting ( sic )' successful re-entry is \% 

that of the expectation of school administrators for the behavior of the returnee.... 
Parolees are required to fulfill* standards of conduct which may be unrealistic for 
even the most successful of students.'* 



What can be seen in the preceeding* discussion illustrates* the difficulty of 
. utilizing student outcome data- either short-term or long-term to illustrate 
program success 1 in correctional facilities. 

The barriers to effective utilization of testing in evaluation are great. The 
barriers to gathering useful follow-up is also .great. It, is not likely that 
either of thes^ approaches alone to evaluate .programs is sufficient to provide 
useful data. r ^ > / 

The prime audience for tjie evaluative reports to be produce was originally con- 
sidered to' be the Federal Government. # The unit has utilized the federal guide-', 
lines as criteria for making judgements about annual report questionnaire data ^ 
collected once a year from each of the projects.* In addition, on-site visits-" 
were conducted to add to the descriptive dimensiop of the reports* Initially, 
reports were focused on tabulor data depicting various dimensions of the proj- 
ects, including student success. Since 1972, however, descriptions moved away 
from tabulor presentation into more narrative styles. 

, » * , * « - 

Data in the 1974 report 'included ''snapshots," or brief capsulations of projects.. 
Further, one site was described in the manner of a "p.ortrayal"- (the techniqufe of 
using -portrayal in evaluative reports has been explored at Center for Instruction 
and Curriculum Evaluation at the University of, Illinois) in a separate report 
(Hanna City). Both "snapshots" and "the portrayal" were constructed in descrip- 
tive narrative style. The snapshots were four or five brief paragraphs long, and 
the more extensive portrayal was seventeen pages long. The 'purpose of the snapshots 
was to allow the reader, to gain some understanding of the issues, conditions , . and 
feelings which existed' in respect' to Title I 89-750 projects.. The extensive pro- 
trayal was constructed to allow readers of the report', to make their own judgements 
as to the value of the Title I 89-750 project. . 



Conclusion 



The data collected by the evaluation unit, for 'the annual reports are intrusions into 
the operation of Title I 8-9-750 projects. The, power of such evaluative interven- 
tion should be recognized. It is necessary to understand that the primary audience 
for evaluative data should be the projecjf itself . Although the federal Require- 
ment of objective measures of student gains canriot'be denied, it must be 'recog- 
nized as inappropriate .for short-term' projects such as Titled! 39-750 Migrant and 
Neglected/Delinquent. .As it* is immediately beyond the 'power of ,$he' Ilfirto is * Evalua- 
tion Unit to change inappropriate phrasing in federal guidelines, it can only' attempt 
to influence local efforts to utilize^ practices'^ that are as-sbuqji as possible. 

Recommendations in reports by the evaluation unit have been aitne<# to compliance wifh 
federal requirements by suggesting. the best practices preferred. Projects need to 
be able to write objectives in appropriate formats and they needed "objective mea- ' 
sutes" of student success. * 1 • t * , • 

-The evaluation unit, in providing a~ r^ange of choices on the LEA repprt questionnaire 
including standardized tests ,' criterion' referenced tests,' observation, ,and yther > 
methods' for projects to. select from fn ascertaining, student success have alerted 
projects to ia range of possibilities. 'Many projects now utilize multiple methods/ 
and criterion referenced methods are increasingly utilized. v * 

The evaluation unit, further, can attempt to .'caution projects to- •use* tests that 
are appropriate considering the nature of the project./ * " / 



,The v "objective meas,ure M approach to evaluation, .although in compliance with 
federal guidelines, has little relationship to project improvement.* For example, 
when questionnaires asked to vhat they attributed student success neglected/ 
delinquent projects personnel typically reported that it was the good project. 
When asked to vhat they attributed lack of success, project personnel typically 
reported factors relating to the students. We could not Expect otherwise if we 
consider evaluation only to be a report of student ^scores. 

The next step suggested by the evaluation unit was the collection of longitudinal, 
information. £s this was or^ly a suggestio^S^nd not a requirement the *level of 
intrusion is small if not non-existant . Future requests for data from projects- 
couIH include sucH a study,, or the evaluation u52t\ as an active-external data 
collector, could gather -'sxich data on its own. * * * 

• * , • « > 

In fact, the role pf the evaluation unit has been to supplement data reported to 
the State* Office of Education through activities of its own. ^The efforts of the 
evaluation unit have been to use N data, gathered from a variety ! pf sources in a 
variety of ways. Survey data, teacher judgements, student judgements, and exter- 
nal observer judgements have been, qsed to add perspectives on the implementation 
of programs. Data 'such as these have been combined into case studies snapshots* 
and portrayals. 

Reports of this type' are .used to' provide projects with perspective on their, own 
activities and the activities of other projects. An additional' s tep in this • - 
approach, riot yet taken by the evaluation unit;, could be to provide such descrip- 
tions to panels of review of relevant audiences or expert juries, f,or 'comments. 
The comments could then be appended to the original document to be reviewed by' * 
project personnel in decision making decisions, s^out how to improve their project. ' 

These efforts of the evaluation unit ifc the ^Hinois Office of Education have been 
to* broaden the nature and scope of evalt^a&^dn as practiced at the project level and 
as- supplemented by efforts by the evaluattLfcjb unit. Eval^tion for Title I 89-750* 
Migrant £*nd Titje I 89-7-50. Neglected/Delin^uent^inust have payoff at' the local 
level. Any data' collection that does not do r so can on£y be viewed as a kind of 
imperialism. " ' * 
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